Erin Bennett, Judith Degen, MH Tessler, Justine Kao & Noah D. Goodman (Stanford)

outline

Bayesian models of cognition

  • may require specifications of subjective beliefs

 

new experimental measure

  • binned histogram task (Kao et al., 2014)

 

Bayesian data analysis

  • great tool to specify and test ideas about how unobservables (like beliefs) relate to each other and how they generate observable data

beliefs are latent

relevance

  • belief + desire + rationality \(\Rightarrow\) action

problem

  • beliefs are latent, only action is directly observable

 

JonesUmbrella

focus: 1d continguous quantities

context

Joe is an adult male from Texas. He went to a BBQ party yesterday.

sentence

Joe ate many burgers.

 

cardinal surprise reading

Joe ate more than expected.

 

 

how to determine beliefs?

real-world frequencies

  • may not exist
  • subjects may not be aware

 

experimental measures

  • give-number task & inference of parameterized distribution
    • e.g., Manski (2004); Tauber & Steyvers (2013)
    • scoring rules (Savage, 1971)
  • iterated learning tasks (Lewandowsky, Griffiths, & Kalish, 2009)
  • binned histogram task (Kao et al., 2014)

experiment

overview

  • 50 participants recruited via MTurk
  • each saw every condition of every task
  • 8 items (from previous research)
  • 3 task types:
    • BH: binned histogram
    • GAN: give-a-number
    • PC: paired comparison

items

  1. "X has just fetched himself a cup of coffee from the office vending machine."
    • "What do you think the temperature of his coffee is?"
  2. "X commuted to work yesterday."
    • "How many minutes do you think she spent commuting yesterday?"
  3. "X told a joke to N kids."
    • "How many of the kids do you think laughed?"
  4. "X bought a laptop."
    • "How much do you think it cost?"
  5. "X threw N marbles into a pool."
    • "How many of the marbles do you think sank?"
  6. "X just went to the movies to see a blockbuster."
    • "How many minutes long do you think the movie was?"
  7. "X watched TV last week."
    • "How many hours do you think he spent watching TV last week?"
  8. "X bought a watch."
    • "How much do you think it cost?"

BH task

priorsslider

dummy

normalize slider rating by subject \(\Rightarrow\) average over subjects \(\Rightarrow\) BH task averages

GAN task

dummy

priorsnumbers

PC task

dummy

priorslightning

results

BH task averages

dataslider

GAN frequencies

dataslider

PC choice proportions

dataslider

Bayesian inference

agenda

goal: scrutinize BH task

  • do BH task averages approximate the central tendency of beliefs in the population?

dummy

approach: hieararchical Bayesian modeling

  • take data from all three tasks
  • infer latent subjective & "population-level beliefs"
  • specify "link functions"
    • how do subjective beliefs generate observable data

dummy

model

modelGraph

set-up

  • implemented in JAGS
  • 50,000 samples after a burn in of 100,000
  • convergence checks: visually and \(\hat{R}\)

population-level beliefs \(Q_j\)

postPriors

red: BH task averages; black: mean posterior \(Q_i\); grey: 95% HDIs

individual vs. population-level beliefs

postSubjPriors

black: mean posterior \(Q_i\); grey: mean posterior \(P_{ij}\)

upshot

 

 

  • subjective beliefs \(P_{ij}\) may differ from population-level mean (good!)

 

  • BH task averages reasonably approximate mean \(Q_j\) (excellent!)

model criticism

PPC averaged normalized slider

ppcSlider

red: observed data; black: mean posterior prediction with 95% HDIs

PPC number choice

ppcNumber

red: observed data; black: mean posterior prediction with 95% HDIs

PPC lightning round

ppcChoice

red: observed data; black: mean posterior prediction with 95% HDIs

conclusions

conclusions

dummy

  • BH task averages are practical and reasonable approximations of the population-level central tendency of individual beliefs

dummy

  • Bayesian data analysis is a great tool to specify and test ideas about how unobservables (like beliefs) relate to each other and how they generate observable data
    • inferences based on diverse data from different tasks

modeling details

model

modelGraph

hierarchical population prior

  • \(w \sim \text{Gamma}(2,0.1)\)
  • \(Q_{j} \sim \text{Dirichlet}(1,\dots, 1)\)
  • \(P_{ij} \sim \text{Dirichlet}(w Q_j)\)

dummy

w = 20

w = 200

link function: BH task

link function: GAN task

link function: PC task

posterior over parameters

postParameters

posterior predictive p-values

posteriorP